Combining video-based and optic-based augmented reality in a near eye display

ABSTRACT

Apparatus, including a retaining structure ( 54 ), positioned near a subject&#39;s eye that has a pupil with a diameter, an optical combiner ( 52 A) mounted on the structure before the eye, and a pixelated screen ( 60 A) having an array of variably transparent pixels coating the combiner. There is an image capturing device ( 68 A) mounted on the structure to capture an image of a scene viewed by the eye, and a projector ( 64 A) is mounted on the structure to project at least one of a portion of the captured image and a stored image onto a section of the screen at a selected location thereof. A processor ( 26 ) renders the screen section at least partially opaque, selects the section location in response to a region of interest in the scene identified by analysis of the captured image, and determines a dimension of the section in response to the pupil diameter.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/419,023, filed 22 May 2019, which is a continuation of U.S. patentapplication Ser. No. 16/159,740, filed 15 Oct. 2018 (now U.S. Pat. No.10,382,748), which is a continuation of U.S. patent application Ser. No.15/896,102, filed 14 Feb. 2018 (now U.S. Pat. No. 10,134,166), which isa continuation of U.S. patent application Ser. No. 15/127,423, filed 20Sep. 2016 (now U.S. Pat. No. 9,928,629), in the national phase of PCTPatent Application PCT/IB2016/051642, filed 23 Mar. 2016, which claimsthe benefit of U.K. Patent Application GB1504935.6, filed 24 Mar. 2015,each of which is incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to a near eye display system,and specifically to a display able to combine video-based andoptic-based augmented reality.

BACKGROUND OF THE INVENTION

A near eye display system may be used in an augmented reality situation,where a scene that is being viewed by a user of the assembly is altered,typically by being augmented or supplemented. The alteration is computerprocessor generated, and typically involves presenting real time video,and/or non-real time images, to the user while the user is gazing at thescene.

U. S. Patent Application 2010/0149073, to Chaum et al., whose disclosureis incorporated herein by reference, describes a near eye displaysystem. The system includes a source of modulated light, and a “proximaloptic” positionable adjacent to an eye of a system user to receive themodulated light. The proximal optic has a plurality of groups ofoptically redirecting regions.

U. S. Patent Application 2012/0068913, to Bar-Zeev et al., whosedisclosure is incorporated herein by reference, describes an opticalsee-through head-mounted display device. The device includes asee-through lens which combines an augmented reality image with lightfrom a real-world scene, while an opacity filter is used to selectivelyblock portions of the real-world scene so that the augmented realityimage appears more distinctly.

U. S. Patent Application 2013/0050258, to Liu et al., whose disclosureis incorporated herein by reference, describes a see-throughhead-mounted display device that provides an augmented reality imagewhich is associated with a real-world object. Initially, the object isidentified by a user, e.g., based on the user gazing at the object for aperiod of time, making a gesture such as pointing at the object and/orproviding a verbal command.

Documents incorporated by reference in the present patent applicationare to be considered an integral part of the application except that, tothe extent that any terms are defined in these incorporated documents ina manner that conflicts with definitions made explicitly or implicitlyin the present specification, only the definitions in the presentspecification should be considered.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides apparatus, including:

-   -   a retaining structure, configured to be positioned in proximity        to an eye of a subject, the eye of the subject having a pupil        with a pupil diameter;    -   an optical combiner mounted on the structure in front of the        eye;    -   a pixelated screen, having an array of variably transparent        pixels, coating the optical combiner;    -   at least one image capturing device mounted on the structure        configured to capture an image of a scene viewed by the eye;    -   a projector mounted on the structure and configured to project        at least one of a portion of the captured image and a stored        image onto a section of the screen at a selected location        thereof; and    -   a processor, configured to render the section of the screen at        least partially opaque, to select the location of the section in        response to a region of interest in the scene identified by        analysis of the captured image, and to determine a dimension of        the section in response to the pupil diameter.

The processor may be configured to identify the region of interest inresponse to radiation received by the image capturing device from atleast one marker located at the region of interest.

The apparatus may include a further image capturing device configured toidentify the region of interest in response to received radiation fromat least one marker located at the region of interest. The at least oneimage capturing device may be configured to operate in the visiblespectrum, and the further image capturing device may be configured tooperate in the non-visible spectrum. The apparatus may include aradiator configured to radiate radiation in the non-visible spectrumtowards the region of interest.

In a disclosed embodiment the apparatus includes at least one markerpositioned in proximity to the region of interest, and wherein theprocessor is configured to detect the marker in the captured image so asto identify the region of interest.

In a further disclosed embodiment the processor is configured todetermine an initial pupil diameter in response to the dimension of thesection being set by the subject to occlude an object of known sizewhile the subject gazes at the object in a known ambient lightbrightness. Typically, the processor is configured to determine abrightness of the scene in response to the captured image of the scene,and the processor is configured to determine the pupil diameter inresponse to the initial pupil diameter and the brightness of the scene.

In a yet further disclosed embodiment the processor is configured todetermine an initial pupil diameter in response to analysis of areflected image of the subject while the subject gazes into a mirror ina known ambient light brightness. Typically, the processor is configuredto determine a brightness of the scene in response to the captured imageof the scene, and the processor is configured to determine the pupildiameter in response to the initial pupil diameter and the brightness ofthe scene.

In an alternative embodiment the dimension of the section is determinedso that the region of interest is occluded. Typically, a regionsurrounding the region of interest is partially occluded. A fraction ofocclusion in the region surrounding the region of interest may bedetermined in response to the pupil diameter. The processor may beconfigured to derive from the captured image an image corresponding tothe region surrounding the region of interest, and the projector may beconfigured to project the derived image onto an area of the screensurrounding the at least partially opaque section of the screen. Anintensity of the projected derived image may be determined in responseto the fraction of occlusion.

In a further alternative embodiment the dimension of the section isdetermined in response to a size of the region of interest.

The dimension of the section may be determined so that an area greaterthan the region of interest is occluded. Alternatively, the dimension ofthe section may be determined so that an area less than the region ofinterest is occluded.

In a yet further alternative embodiment the retaining structure is aspectacle frame. Alternatively, the retaining structure is a helmethaving a head-up display.

Typically, the at least one image capturing device includes two imagecapturing devices capturing respective images of the scene, and theprocessor is configured to identify the region of interest by analysisof the respective images.

There is further provided, according to an embodiment of the presentinvention, a method, including:

-   -   positioning a retaining structure in proximity to an eye of a        subject, the eye of the subject having a pupil with a pupil        diameter;    -   mounting an optical combiner on the structure in front of the        eye;    -   coating the optical combiner with a pixelated screen, having an        array of variably transparent pixels;    -   mounting at least one image capturing device on the structure so        as to capture an image of a scene viewed by the eye;    -   mounting a projector on the structure the projector being        configured to project at least one of a portion of the captured        image and a stored image onto a section of the screen at a        selected location thereof;    -   rendering the section of the screen at least partially opaque;    -   selecting the location of the section in response to a region of        interest in the scene identified by analysis of the captured        image; and determining a dimension of the section in response to        the pupil diameter.

There is further provided, according to an embodiment of the presentinvention, apparatus, including:

-   -   a retaining structure, configured to be positioned in proximity        to an eye of a subject;    -   an optical combiner mounted on the structure in front of the        eye;    -   a pixelated screen, having an array of variably transparent        pixels, coating the optical combiner;    -   at least one image capturing device mounted on the structure        configured to capture an image of a scene viewed by the eye;    -   a processor, configured to render a section of the screen at        least partially opaque, and    -   a projector mounted on the structure and configured to project        at least one of a portion of the captured image and a stored        image onto the section of the screen so that there is        misalignment between the scene viewed by the eye through the        combiner and the at least one portion of the captured image and        the stored image.

Typically, for a scene at 50 cm from the eye, the misalignment is nomore than 2 cm.

The projector may be configured to project the portion of the capturedimage and the stored image, in registration with each other, onto thesection of the screen.

There is further provided, according to an embodiment of the presentinvention, apparatus, including:

-   -   a retaining structure, configured to be positioned in proximity        to an eye of a subject;    -   an optical combiner mounted on the structure in front of the        eye;    -   a rotator connected to the optical combiner and configured to        rotate the optical combiner about an axis;    -   a pixelated screen, having an array of variably transparent        pixels, coating the optical combiner;    -   at least one image capturing device mounted on the structure        configured to capture an image of a scene viewed by the eye; and    -   a processor, configured to render a section of the screen at        least partially opaque, and to activate the rotator so that the        optical combiner is oriented to be orthogonal to a region of        interest in the scene.

The processor is typically configured to select the section of thescreen so as to occlude the region of interest.

The axis may be a vertical axis.

There is further provided, according to an embodiment of the presentinvention, a method, including:

-   -   positioning a retaining structure in proximity to an eye of a        subject;    -   mounting an optical combiner on the structure in front of the        eye;    -   coating the optical combiner with a pixelated screen comprising        an array of variably transparent pixels;    -   mounting at least one image capturing device on the structure,        the device being configured to capture an image of a scene        viewed by the eye;    -   rendering a section of the screen at least partially opaque;    -   mounting a projector on the structure; and    -   configuring the projector to project at least one of a portion        of the captured image and a stored image onto the section of the        screen so that there is misalignment between the scene viewed by        the eye through the combiner and the at least one portion of the        captured image and the stored image.

There is further provided, according to an embodiment of the presentinvention, a method, including:

-   -   positioning a retaining structure in proximity to an eye of a        subject;    -   mounting an optical combiner on the structure in front of the        eye;    -   connecting a rotator to the optical combiner, the rotator being        configured to rotate the optical combiner about an axis;    -   coating the optical combiner with a pixelated screen having an        array of variably transparent pixels;    -   mounting at least one image capturing device on the structure,        the device being configured to capture an image of a scene        viewed by the eye;    -   rendering a section of the screen at least partially opaque; and    -   activating the rotator so that the optical combiner is oriented        to be orthogonal to a region of interest in the scene.

The present disclosure will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates use of an augmented reality system,according to an embodiment of the present invention;

FIGS. 2A-2E are schematic diagrams illustrating an augmented realityassembly, as well as functions that may be implemented in the assembly,according to an embodiment of the present invention;

FIGS. 3A, 3B, and 3C are schematic diagrams illustrating the assembly indifferent orientations with respect to a region of interest, and FIGS.3D and 3E are graphs derived from the different orientations, accordingto an embodiment of the present invention;

FIG. 4 is a schematic diagram illustrating derivation of the dimensionsof an occlusion mask, according to an embodiment of the presentinvention;

FIG. 5 is a schematic diagram illustrating partial occlusion of an areaaround the region of interest, according to an embodiment of the presentinvention;

FIG. 6 illustrates an occlusion mask, according to an embodiment of thepresent invention;

FIG. 7 illustrates graphs of occlusion vs. distance, according to anembodiment of the present invention; and

FIG. 8 is a flowchart of steps performed in operation of the augmentedreality system, according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

An embodiment of the present invention provides a near eye assemblyhaving a retaining structure that is configured to be positioned inproximity to the eye of a user of the assembly. Typically, the retainingstructure comprises a spectacle frame. Alternatively, the retainingstructure comprises a head up-display which may be mounted on a helmetworn by the assembly user.

An optical combiner is mounted on the structure in front of the usereye. Typically, two combiners are mounted, one in front of each eye. Theoptical combiner at least partially transmits elements of a scene infront of the assembly through the combiner. In addition, the opticalcombiner may receive a visible radiation transmission derived from ascene, and/or a visual transmission such as a presentation of data or amarker, and redirects the transmission back to the user's eye.

A pixelated screen, comprising an array of variably transparent pixels,coats the optical combiner. Typically, the pixels are liquid crystaldisplay (LCD) pixels.

There is at least one image capturing device, typically two suchdevices, one for each eye, mounted on the structure. The capturingdevice is typically a visible spectrum camera that is configured tocapture an image of a scene viewed by the user's eye.

A projector, typically a micro-projector, is mounted on the structure.Typically two projectors, one for each eye, are mounted on thestructure. The projector is configured to project at least one of aportion of the captured image as a video, as well as a stored image,onto a section of the screen that a processor renders at least partiallyopaque. The at least partially opaque section is also referred to hereinas an occlusion mask, or just as a mask.

The processor is configured to select the location of the section inresponse to a region of interest in the scene identified by analysis ofthe captured image. Typically, at least one marker is positioned nearthe region of interest, and the processor analyzes the captured image tolocate the marker and so identify the region of interest. Rendering thesection opaque occludes the region of interest from the user's eye.

In addition, the processor determines a dimension of the section,typically, in the case of the section being circular, the diameter ofthe section. The dimension is determined in response to the pupildiameter.

By setting the dimension of the section according to the pupil diameter,embodiments of the present invention more exactly control the area ofthe region of interest that is occluded. In addition, because of thefinite size of the pupil, there is a region surrounding region ofinterest that is partially occluded. In some embodiments the processoroperates the micro-projector to overlay relevant portions of thecaptured image on the partially occluded region, so as to compensate forthe partial occlusion.

As stated above, a portion of the captured image may be projected as avideo onto the occlusion mask. In some embodiments the captured imageportion video corresponds to the occluded region of interest. There is anon-occluded region surrounding the occluded region of interest, andthis non-occluded region is visible to the user through the combiner. Inembodiments of the invention the video and the visible non-occludedregion are typically not in accurate registration, due to slightinevitable movements of the display relative to the user's eye.

In some embodiments a stored image, such as an image of a tool, isoverlaid on, and in accurate registration with, the occluded regionvideo.

The inventors have found that registering the stored image with thevideo, even though the video is not fully registered with thesurrounding visible region, provides an acceptable image for the user.The inventors have found that for a non-occluded region that appears tobe 50 cm from the user's eye, the video and the non-occluded region maybe out of registration by up to 2 cm, while still being acceptable tothe user.

Thus, in contrast to prior art augmented reality systems, embodiments ofthe present invention are configured to operate with mis-alignmentbetween the visible portion of a scene and an augmented reality portionof the scene. However, there is no mis-alignment between elements withinthe augmented reality video, i.e., the elements projected onto theocclusion mask.

In some embodiments, the optical combiner may be rotated about an axisby the processor. In the case of two combiners, they may beindependently rotated about respective axes. The independent rotationsmay be used to orient both combiners so that each is orthogonal to thedirection of gaze of the user's eyes.

SYSTEM DESCRIPTION

Reference is now made to FIG. 1 , which schematically illustrates use ofan augmented reality system 20, according to an embodiment of thepresent invention. By way of example and for simplicity, in thefollowing description system 20 is assumed to be used in a medicalprocedure during part of which the user of the system is being mentored.However, it will be understood that embodiments of the present inventionmay be used in non-medical and/or non-mentoring situations, such as inoperating a video game, in simulating a real-world event, or inproviding an aid to navigation.

System 20 is operated by a medical professional 22, who wears anaugmented reality assembly 24, described in more detail below withrespect to FIGS. 2A-2D. While assembly 24 may be incorporated forwearing into a number of different retaining structures on professional22, in the present description the retaining structure is assumed to besimilar to a pair of spectacles. Those having ordinary skill in theaugmented reality art will be aware of other possible structures, suchas incorporation of the augmented reality assembly into a head-updisplay that is integrated into a helmet worn by the user of system 20,and all such structures are assumed to be comprised within the scope ofthe present invention.

System 20 comprises and is under overall control of a processor 26. Inone embodiment processor 26 is assumed to be incorporated within astand-alone computer 28, and the processor typically communicates withother elements of the system, including assembly 24, wirelessly, as isillustrated in FIG. 1 . Alternatively or additionally, processor 26 mayuse optical and/or conducting cables for communication. In furtheralternative embodiments processor 26 is integrated within assembly 24,or in the mounting of the assembly. Processor 26 is typically able toaccess a database 40, wherein are stored images and other visualelements used by system 20. Software enabling processor 26 to operatesystem 20 may be downloaded to the processor in electronic form, over anetwork, for example. Alternatively or additionally, the software may beprovided on non-transitory tangible media, such as optical, magnetic, orelectronic storage media.

The medical procedure exemplified here is on a patient 30, and duringthe procedure professional 22 gazes along gaze directions 32 at a regionof interest (ROI) 34. ROI 34 typically, but not necessarily, comprises aportion of the patient. In some embodiments one or more ROI acquisitionmarkers 35, comprising marker elements 36, are positioned in, and/or inproximity to, ROI 34, and the functions of such markers are describedbelow. Typically there are at least three marker elements 36 for a givenmarker 35. In a disclosed embodiment the size of ROI 34 may bepredefined by professional 22, for example based on a computerizedtomography (CT) image of the patient, and the position of the ROI mayalso be a predefined distance to the right and a predefined distancebelow the marker. In an alternative embodiment marker elements 36 ofmarker define ROI 34 to be a region within a surface having elements 36in the perimeter of the surface. Typically, a margin in an approximaterange of 1-5 cm is added to ROI 34 to compensate for mis-alignmentbetween a video projection and a directly viewed scene, described inmore detail below.

During the procedure professional 22 may use a surgical device 38, suchas a surgical knife, to perform part of the procedure. Typically device38 comprises one or more identifying elements 39 which may be used totrack the device.

FIGS. 2A-2E are schematic diagrams illustrating assembly 24, as well asfunctions that may be implemented in the assembly, according to anembodiment of the present invention. FIG. 2A illustrates assembly 24with none of the active elements of the assembly, i.e., those elementsrequiring power, operating. As stated above, assembly 24 is configured,by way of example, as a pair of spectacles 50. Similar elements of each“half” of the pair of spectacles are referred to generically by anidentifying numeral, and the similar elements are differentiated asnecessary by adding a letter to the numeral. Thus spectacles 50 compriseplanar optical combiners 52, comprising combiners 52A and 52B in frontof, respectively, the left and right eyes of professional 22. Opticalcombiners 52 are mounted on a retaining structure 54 which holdselements of assembly 24, and which is herein assumed to comprise aspectacle frame, so that structure 54 is also referred to herein asframe 54.

In some embodiments, combiner frames 82A and 82B are fixed to retainingstructure 54 and vertical retaining rods 84A and 84B attached to thecombiner frames support the optical combiners, so that the combiners areable to rotate about vertical axes defined by the rods. Retaining rods84A and 84B, and thus combiners 52A and 52B, may be rotatedindependently of each other about their vertical axes by respectivemotors 86A and 86B, fixed to frames 82A and 82B. Motors 86, typicallystepper motors, are controlled by processor 26 so as to rotate theirattached combiners to known, typically different, fixed orientationswith respect to their respective combiner frames.

Each optical combiner 52 is configured to at least partially transmitelements of a scene through the combiner, so that a portion 56 ofpatient 30 (FIG. 1 ) is assumed to be directly visible through eachcombiner 52. In addition, each optical combiner 52 is configured toreceive a visible radiation transmission derived from a scene, and/or avisual transmission such as a presentation of data or a marker, and toredirect or reflect the transmission back to the eye of professional 22.The redirection is such that the scene and/or data or marker presentedto the professional appears to be at a distance between the near and farpoints of vision of the professional. Thus, any given section of theoptical combiner may combine directly visible material with redirectedor reflected material, and provide this combined material to the eye ofthe user. More detail of the functioning of combiners 52 is providedbelow.

Optical combiners of various types are known in the art. One known typeuses a semi reflective surface which transmits an image from an imagesource after it has passed through a set of lenses which correctdeformations caused by the semi reflective surface of the combiner.Another known type uses a waveguide which projects the image directly tothe eye of the viewer. Herein, by way of example, combiners 52 areassumed to be of the waveguide type.

In one embodiment, combiners 52 comprise LUMUS DK 32 see throughglasses, produced by Lumus Optical of Rechovot, Israel.

Generally similar pixelated variable transparency screens 60A and 60Brespectively coat a rear side, i.e., the side away from the eyes ofprofessional 22, of combiners 52A, 52B. Screens 60 are active elementsof system 20 and are formed of an array of pixels, the opacity of eachof the pixels being controlled by processor 26.

Screens 60 are typically, but not necessarily, liquid crystal displays(LCDs) formed of a rectangular array of liquid crystal pixels.Alternatively, screens 60 are formed of MEMS (microelectromechanicalsystems). Further alternatively, screens 60 are formed of polymerdispersed liquid crystals (PDLCs). In the following description, by wayof example, screens 60 are assumed to be formed of LCDs. LCD displaypixels can typically be switched between an opaque state, whereapproximately 95% of the incoming light is blocked and 5% istransmitted, and a transparent state where approximately 60% of theincoming light is blocked and 40% is transmitted. The LCDs then have atransmission contrast ratio of 1:8.

Fixedly attached to arms of frame 54 are generally similarmicro-projectors 64A and 64B. Each micro-projector is located andoriented so as to be able to project onto respective combiner 52A and52B, a scene, and/or a visual indication, in a form suitable forredirection by the combiners to the left or right eye of professional22. Micro-projectors 64 are active elements, and the projectedscenes/indications are provided to the micro-projectors by processor 26.The projection and redirection are configured so that the images seen bythe eyes of professional 22, absent any correcting lenses, appear to beat infinity, due to parallel light coming from the combiners andentering the pupils. In some embodiments display 24 comprises correctinglenses 88A, 88B which redirect light from combiners 52A, 52B so that theimages appear to be closer than infinity to the professional's eyes. Thepower D in diopters of the lenses defines the distance d of the images,according to the formula d=1/D, where d is in meters, and D is anegative number. Lenses 88A, 88B are typically located between theprofessional's eyes and the respective combiners. For simplicity, lenses88A, 88B are not shown in other figures of the present application.

At least one image capturing device 68 is attached to frame 54. In thedisclosed embodiment there are two generally similar devices 68A and68B, respectively aligned to be approximately orthogonal to planarcombiners 52A and 52B, so as to be able to capture radiation ofrespective images of scenes viewed by the left and right eyes ofprofessional 22. Typically, devices 68 comprise cameras configured tocapture images of scenes in the visible spectrum. The cameras may userolling shutters, in which cases latency (of projection viamicro-projectors 64) may be reduced by processing rows of images ratherthan complete frames of images. In some embodiments devices 68 may alsocapture non-visible portions of images, such as portions in theinfra-red spectrum. The operation of devices 68 is controlled byprocessor 26.

In some embodiments of the present invention, assembly 24 comprises asensor 72 which is configured to capture non-visible images of elementsof a scene in front of assembly 24. Typically sensor 72 uses a projector73 configured to project radiation in the non-visible spectrum detectedby the sensor, and has a bandpass filter configured to block visibleradiation, such as that projected by surgical lighting. Typically,sensor 72 and projector 73 operate in the near infra-red spectrum.

In some embodiments, assembly 24 comprises a manual and/or electroniccontrol 74 which may be operated by professional 22 to move elements ofthe assembly in and out of the field of view of the professional.Additionally or alternatively, there may be a button or switch 78 whichenables the professional to power active elements of assembly 24, suchas the capturing devices and the micro-projectors. In some embodimentsswitch 78 may be a foot switch. Further additionally or alternatively,assembly 24 may be configured so that it can tilt downwards about ahorizontal axis, at an angle up to 400 from the horizontal, so that theprofessional can look through the assembly when looking down.

Additionally, assembly 24 may comprise a sensor 76, such as anaccelerometer, which is configured to measure an inclination of theassembly with respect to the direction of gravity, so measuring theangle of the head of the professional with respect to the vertical.Processor 26 may be configured to use readings from sensor 76 to moveelements of assembly 24 in and out of the field of view of theprofessional, and/or to control whether micro-projectors 64 projectimages.

FIG. 2B schematically illustrates the appearance of assembly 24 whenprocessor 26 activates screens 60A and 60B. As described above, eachscreen 60 comprises an array of pixels, and the opacity of each pixel inan individual screen may be set by processor 26. In screen 60A theprocessor has rendered a circular array 80A of the pixels of the screenopaque, while the remaining pixels of the screen are renderedtransparent. The opacity of array 80A means that from the point of viewof the left eye of professional 22, circular array 80A acts as a maskoccluding corresponding features of portion 56 of the patient, so thatarray 80A is also referred to herein as occluding mask 80A.

Similarly in screen 60B processor 26 has rendered a circular array 80Bof the pixels of the screen opaque, while the remaining pixels of thescreen are rendered transparent. As for array 80A, array 80B occludessections of portion 56 from the view of the right eye of professional22. Thus array 80B is also referred to herein as occluding mask 80B.

FIG. 2C schematically illustrates the appearance of assembly 24 whenprocessor 26 activates screens 60 and micro-projectors 64A and 64B.Screens 60 are activated to provide occluding masks 80A and 80B, asdescribed above with respect to FIG. 2B. Micro-projector 64A projects aprerecorded ultrasound image 90A of the patient's abdomen so as tooverlay the image on mask 80A, and micro-projector 64B projects an image90B of the abdomen so as to overlay it on mask 80B. Typically, althoughnot necessarily, images 90A and 90B are the same. In some cases, forexample if the images have been acquired in a stereoscopic manner or forcorrect 3D perception, images 90A and 90B may be slightly different,typically being slightly displaced horizontally with respect to eachother. Micro-projectors 64 are configured to position images 90A and 90Bon their respective masks so that, as seen by professional 22 and with−2 diopter lenses 88A, 88B present, the images are in focus atapproximately 50 cm and appear to be at the location of the patient'sabdomen.

In addition to projecting images 90, micro-projectors 64 also projectalphanumeric data 92A and 92B onto the non-occluded region of screens60, as well as markers 96A and 96B onto masks 80A and 80B. Images 90,data 92, and markers 96 are typically stored in database 40, and areprovided from the database to micro-projectors 64 by processor 26.

In a mentoring situation images 90, the contents of data 92, and theposition of markers 96 are typically under control of a tutorinteracting with processor 26 while mentoring professional 22. In somecases the locations of masks 80 may also be provided to processor 26 bythe tutor, although typically the locations of the masks depend upongaze directions 32 of the professional. In a non-mentoring situation,i.e. where professional 22 alone operates system 20, locations of masks80 are typically automatically set by processor 26, as is describedbelow. Also in a non-mentoring situation, images 90, data 92, andmarkers 96 may be controlled by professional 22. It will be understoodthat images 90, data 92 and markers 96 are examples of non-video relatedvisual elements that are seen by professional 22, and that the provisionof such elements corresponds to an optic-based augmented realitysituation implemented in system 20.

FIG. 2D schematically illustrates the appearance of assembly 24 whenprocessor 26 activates screens 60 and micro-projectors 64, and inaddition incorporates a video-based augmented reality feature into theoperation of the assembly. Screens 60 and micro-projectors 64 areassumed to be activated as described above for FIG. 2C, so that masks80, images 90, data 92 and markers 96 are in the field of view ofprofessional 22. By way of example, the figure has been drawn toillustrate a mentoring situation, where the tutor of professional 22wants to point to a feature of the chest of patient 30, herein assumedto comprise an unusual movement of the chest.

To point to the feature, the tutor interacts with processor 26 so thatthe processor enhances and emphasizes portions 100A, 100B of the videoimages acquired by capturing devices 68, the portions corresponding tothe region of the chest where the unusual movement is occurring.Micro-projectors 64A, 64B then project portions 100A, 100B ontocombiners 52A, 52B. It will be understood that the enhancement ofportions 100A, 100B and their projection on the respective combiners isin real-time. The enhancement may take a number of forms. For example,portions 100A, 100B may comprise a wireframe image of the region of thechest having unusual movement, and/or a false-color image of the region.Other suitable methods of real-time enhancement will be apparent tothose having ordinary skill in the art, and all such methods are assumedto be within the scope of the present invention.

FIG. 2E schematically illustrates an overall scene 101 as seen byprofessional 22, during an invasive surgical procedure being performedby the professional. For simplicity, FIG. 2E illustrates the scene as itis presented on combiner 52A, and it will be understood that asubstantially similar scene is presented to the professional on combiner52B. The figure illustrates a hand 102 of professional 22 holding device38, herein assumed to comprise a pair of tweezers, at a proximal end ofthe device. One or more device identifying elements 39, typicallyreflectors and/or radiators, are attached to the tweezers, so thatprocessor 26 is able to identify and track device 38 using imagesacquired by capturing devices 68.

The professional has made an incision 104 in a portion 106 of patient30, and ROI 34, defined by marker elements 36, is assumed to be at thelocation of the incision. In addition, the professional has inserted alower portion of the distal end of device 38 into the patient so thatthe lower portion is no longer visible.

Processor 26 has formed mask 80A on combiner 52A so as to occlude ROI34, and the portion of incision 104 comprised in the ROI. Mask 80A alsoincludes a margin 83, typically corresponding to a margin ofapproximately 1-5 cm at the ROI. Thus, all elements of the scene outsidemask 80A, comprising hand 102 and the proximal end of device 38, aredirectly visible through combiner 52A by the professional. However,elements of the scene within mask 80A, including a portion of incision104 and an upper portion of the distal end of device 38 that is outsidethe patient, are not visible to the professional, since they areoccluded by the mask.

Processor 26 overlays on mask 80A a captured image 110 of the ROI andthe region corresponding to margin 83, which includes the portion ofincision 104 occluded by the mask and which also includes a video image114 of the upper portion of the distal end of device 38 (outside thepatient) that has been captured by image capturing device 68. Inaddition, the processor overlays on the occlusion mask a stored image112 corresponding to the lower portion of the distal end of device 38(within the patient). Stored image 112 is a virtual elongation of image114 and is retrieved from database 40. The section of the distal endcorresponding to image 112 is not visible to capturing device 68.

The processor registers the two overlaid images, image 110 and image112, with each other, and the registration is possible since by trackingdevice 38 the processor is aware of the location of the device distalend with respect to the captured image. Thus, there is no misalignmentbetween stored image 112, corresponding to the lower portion of thedistal end, and image 114 of the upper portion of the distal end, whichis included in captured image 110.

However, there is typically misalignment between the two registeredoverlaid images 110, 112 and the directly visible portion of scene 101,including the directly visible portion of incision 104, as isillustrated in the figure. The misalignment occurs because while thecaptured image of the ROI is close to that seen by the professional (inthe absence of the occlusion mask), it is not exactly in registrationwith the viewed scene. The inventors have found that a misalignment ofup to 2 cm, in a scene that is 50 cm from the eye of the professional,is acceptable.

FIGS. 3A and 3B are schematic diagrams illustrating assembly 24 indifferent orientations with respect to ROI 34, FIG. 3C is a schematicdiagram illustrating angles of the assembly for the differentorientations, and FIGS. 3D, 3E are graphs of the angles, according to anembodiment of the present invention. For simplicity, combiner frames 82are not shown in the diagrams. In FIGS. 3A and 3B processor 26 haspositioned masks 80 so that they act to occlude ROI 34 from eyes 120A,120B of professional 22, specifically from pupils 124A, 124B of theprofessional's eyes. FIG. 3A illustrates a situation where ROI 34 isapproximately directly in front of professional 22. In this case theprocessor forms mask 80A′ to be on a straight line with pupil 124A andROI 34, while simultaneously forming mask 80B′ to be on a straight linewith pupil 124B and the region of interest.

FIG. 3B illustrates a situation where ROI 34 is not directly in front ofprofessional 22, but is towards one side of the professional. As for thesituation of FIG. 3A the processor forms mask 80A″ to be on a straightline with pupil 124A and ROI 34, and forms mask 80B″ to be on a straightline with pupil 124B and the region of interest. In all cases masks 80act as occlusion masks, and, as is illustrated by the differences in thepositions of the masks, the processor changes the locations of the masksto account for changes in orientation of the region of interest withrespect to assembly 24.

A ring 130 surrounding ROI 34 is described in more detail below.

FIG. 3C schematically illustrates the two situations of FIGS. 3A and 3B,when ROI 34 is at a distance L from eyes 120A, 120B of the professional.Eyes 120A and 120B are separated by a distance w. For the situation ofFIG. 3A, where the region of interest is directly in front of theprofessional, ROI 34 is at a location 103. For the situation of FIG. 3B,ROI 34 is to the left of the professional, at a location 104 that is adistance R from location 103.

For the first situation, where professional 22 is looking at location103, the directions of gaze, α_(R), α_(L) of the professional are shownby lines 103R and 103L. α_(R), α_(L) are angles that are measured withrespect to lines orthogonal to a line connecting eyes 120A, 120B, andtheir values are given by the following equations:

$\begin{matrix}{{\alpha_{L} = {{- {arc}}{\tan\left( \frac{w}{2L} \right)}}},{\alpha_{R} = {{+ {arc}}{\tan\left( \frac{w}{2L} \right)}}}} & (A)\end{matrix}$

For the first situation processor 26 rotates combiners 52A and 52B (forclarity the combiners are not shown in the figure for the firstsituation), within their respective frames 80A and 80B, so that they areorthogonal to lines 103L and 103R. Thus the orientation of the combinersto their frames is given by equations (A).

For the second situation, where professional 22 is looking at location105, the directions of gaze of the professional are shown by lines 105Land 105R. These directions are respectively changed from the “straightahead” directions by β_(L), β_(R). The values of β_(L), β_(R) are givenby equations (B):

$\begin{matrix}{{\beta_{L} = {a{\cos\left( \frac{\sqrt{\left( \frac{w}{2} \right)^{2} + L^{2}} - \frac{R}{\sqrt{1 + \frac{4L^{2}}{w^{2}}}}}{\sqrt{\left( {R - \frac{w}{2}} \right)^{2} + L^{2}}} \right)}}}{\beta_{R} = {a{\cos\left( \frac{\sqrt{\left( \frac{w}{2} \right)^{2} + L^{2}} + \frac{R}{\sqrt{1 + \frac{4L^{2}}{w^{2}}}}}{\sqrt{\left( {R + \frac{w}{2}} \right)^{2} + L^{2}}} \right)}}}} & (B)\end{matrix}$

For the second situation processor 26 rotates combiners 52A and 52B,within their respective frames 80A and 80B, so that they are orthogonalto lines 105L and 105R. Thus the orientation of the combiners to theirframes is given by equations (B), and these orientations are illustratedin the figure.

FIG. 3D is a graph of angles β_(L), β_(R) vs. R for values of L=512 mm,w=60 mm. FIG. 3E is a graph of absolute angles γ_(L), γ_(R), of theangles made by combiners 52A, 52B with their frames where

γ_(L′)=β_(L)+α_(L),γ_(R)=β_(R)+α_(R)  (C)

From the above equations, as well as from the graphs, it is apparentthat the angles made by combiners 52A, 52B with their respective framesare different, as professional 26 gazes at a region of interest. Inaddition, if the professional changes his/her gaze, the changes of thecombiner angles to maintain orthogonality with the gaze directions arealso different.

It will be understood that calculations based on equations herein,including equations (A), (B), and (C), assume that combiners 52A, 52Btransmit rays that are orthogonal to the combiners. Those havingordinary skill in the art will be able to adapt the calculations,mutatis mutandis, for situations where the combiners transmitnon-orthogonal rays.

FIG. 4 is a schematic diagram illustrating derivation of the dimensionsof occlusion mask 80, according to an embodiment of the presentinvention. FIG. 4 is derived from a section of FIG. 3B, specifically thesection illustrating eye 120A, with its pupil 124A, being occluded bymask 80A″ while the eye is gazing at ROI 34. FIG. 4 illustrates across-section of the eye, the occlusion mask, and the region ofinterest. The figure has been drawn on xy axes with an origin O of theaxes at the center of pupil 124 and the directions of the x and y axesbeing respectively orthogonal to and in the plane of the pupil. Mask80A″ and ROI 34 are assumed to be orthogonal to, and to be symmetricallydisposed with respect to, the x-axis. Pupil 124 is assumed to besubstantially circular. For simplicity, mask 80A″ and ROI 34 are alsoassumed to be substantially circular. However, those having ordinaryskill in the art will be able to adapt the following description,mutatis mutandis, for regions of interest and occlusion masks that arenon-circular, so that the scope of the present invention is assumed tocomprise both circular and non-circular regions of interests and masks.

The diagram has been drawn assuming that mask 80A″ just completelyoccludes ROI 34. Thus a ray HB, from an upper edge H of ROI 34 to anupper edge B of pupil 124A touches an upper edge F of mask 80A″.Similarly, a ray GA, from a lower edge G of ROI 34 to a lower edge A ofpupil 124A touches a lower edge E of mask 80A″. Rays HB and GA areassumed to cross at an imaginary point J. A line from upper pupil edge Bparallel to the x-axis cuts mask 80A″ at K and ROI 34 at M.

In the description below:

-   -   p is the apparent diameter of pupil 124A, as measured externally        to eye 120A, corresponding to AB; and    -   d is the diameter of mask 80A″, corresponding to EF; d=d₁ for a        realistic case of p>0, d=d₀ is the diameter of the mask for a        theoretical “pinhole” case of p=0.

In addition,

-   -   D is the diameter of ROI 34 (which is occluded by mask 80A″),        corresponding to GH;    -   L is the distance from pupil 124A to ROI 34;    -   l₁ is the distance from pupil 124A to point J; and    -   l is the distance from pupil 124A to mask 80A″.

In FIG. 4 ΔJFE∥ΔJHG, so that

$\begin{matrix}{\frac{d}{D} = \frac{l + l_{1}}{L + l_{1}}} & (1)\end{matrix}$

From equation (1),

$\begin{matrix}{d = {\frac{l + l_{1}}{L + l_{1}} \cdot D}} & (2)\end{matrix}$

If l₁=0, (for the theoretical case of p=0), then

d=d ₀=1/L·D  (3)

If l₁>0, for the realistic case of p>0, then

$\begin{matrix}{d = {d_{1} = {\frac{l + l_{1}}{L + l_{1}} \cdot D}}} & (4)\end{matrix}$ $\begin{matrix}{{{\Delta{BFK}} \parallel {\Delta{BHM}}},{{{so}{that}\frac{FK}{BK}} = \frac{FM}{BM}}} & (5)\end{matrix}$

For p>0 (so d=d₁) and substituting values of d₁, p, 1, and L for FK, BK,FM, and BM in equation (5) gives:

$\begin{matrix}{\frac{\frac{d_{1}}{2} - \frac{p}{2}}{l} = \frac{\frac{D}{2} - \frac{p}{2}}{L}} & (6)\end{matrix}$

Equation (6) rearranges to:

$\begin{matrix}{d_{1} = {\frac{l\left( {D - p} \right)}{L} + p}} & (7)\end{matrix}$

Equation (7) gives dimensions of mask 80A″, i.e., its diameter d₁, interms of the diameter D of ROI 34, the distance 1 of the mask from thepupil, the diameter of the pupil, and the distance L of the ROI from thepupil.

For typical values of 1=2 cm, L=50 cm, p=0.3 cm, and D=15 cm thediameter of mask 80A″ to just give complete occlusion is, from equation(7), approximately 0.9 cm. For the same values but with p=0.15, the maskdiameter is approximately 0.7 cm.

While, as described above, mask 80A″ completely occludes ROI 34, thereare regions outside ROI 34 that are partly occluded by the mask. Thepartial occlusion follows from the finite, non-zero diameter of thepupil of eye, in the example described here pupil 124A, and is describedin more detail with reference to FIG. 5 below.

FIG. 5 is a schematic diagram illustrating partial occlusion of an areaaround ROI 34, by mask 80A″, according to an embodiment of the presentinvention. FIG. 5 is based upon FIG. 4 , but for clarity some of theelements of FIG. 4 are omitted in FIG. 5 , while other elements areadded in. Thus, a line through point A, parallel to the x-axis, cuts ROI34 at N. A point Q, lying in the same plane as ROI 34, and at a distanceR from the x-axis, is assumed to project two rays—a lower ray 150 whichtouches lower edge A of the pupil, and an upper ray which, but for thepresence of mask 80A″, would touch upper edge B of the pupil. Point Q isthus partly occluded by mask 80A″.

In FIG. 5 at a distance x₁ from the pupil lower ray 150 is assumed to bea distance f1(x₁) from the x-axis, and upper ray 160 is assumed to be adistance f2(x₁) from the x-axis. A line parallel to the y-axis, at x₁,cuts BM at S, AN at T, upper ray 160 at V and lower ray 150 at W. Upperray 160 cuts mask 80A″ at V, and lower ray 150 cuts a plane containingthe mask at W′.

At mask 80A″ the distances of lower ray 150 and of upper ray 160 fromthe x-axis are respectively f1(1) and f2(1), and the width of the beambetween the upper and lower rays is:

f1(1)−f2(1)  (8)

From the diagram,

-   -   partial occlusion occurs if:

f1(1)>d/2 and f2(1)<d/2  (9)

-   -   no occlusion occurs if:

f2(1)≥d/2  (10)

-   -   and full occlusion, corresponding to the situation illustrated        by FIG. 4 , occurs if:

f1(1)≤d/2  (11)

From expressions (8) and (9), and inspection of FIG. 5 , an equation forthe fraction F_(2D) of occlusion occurring is:

$\begin{matrix}{F_{2D} = \frac{\frac{d}{2} - {f2(l)}}{{f1(l)} - {f2(l)}}} & (12)\end{matrix}$

(The subscript 2D indicates that the fraction considered here is for thetwo-dimensional case illustrated in FIGS. 4 and 5 . A fraction for thethree-dimensional case is referred to below.)

Since ΔATW∥ΔΔNQ

$\begin{matrix}{{f1(l)} = {{\frac{l}{L}\left( {R - \frac{p}{2}} \right)} + \frac{p}{2}}} & (13)\end{matrix}$

Since ΔBSV∥ΔBMQ

$\begin{matrix}{{f2(l)} = {{\frac{l}{L}\left( {R + \frac{p}{2}} \right)} - \frac{p}{2}}} & (14)\end{matrix}$

From equations (13 and (14) the diameter of the cone cross-section fromQ at mask 80A″, which is f1(1)-f2(1), is given by:

f1(1)−f2(1)=V′W′=p(1−1/L)  (15)

Substituting equations (14) and (15) into equation (12) gives thefollowing expression for F_(2D):

$\begin{matrix}{F_{2D} = {\frac{V\prime E}{V\prime W\prime} = \frac{\frac{d}{2} - {\frac{l}{L}\left( {R + \frac{p}{2}} \right)} + \frac{p}{2}}{p\left( {1 - \frac{l}{L}} \right)}}} & (16)\end{matrix}$

Inspection of equation (16) indicates that the fraction of occlusion atpoint Q is a function of pupil diameter p, and also decreases linearlyas R increases.

HG is a cross-section of circular ROI 34, so that it will be understoodthat GQ is a cross-section of a circular, partially occluded circularring 130 surrounding ROI 34. As illustrated in FIG. 5 , there is a pointQ′, having the same distance R as Q from the x-axis (but on the oppositeside of the axis), and in the same plane as ROI 34, so that HQ′ is alsoa cross-section of ring 130.

The rays from point Q define a cone of rays emanating from Q, and thiscone cuts mask 80A″ in a circle having a diameter V′W′, the diameterbeing given by equation (15). The cutting of mask 80A″ by the cone ofrays from Q is described with reference to FIG. 6 below.

FIG. 6 illustrates mask 80A″ drawn in a plane orthogonal to the x-axis,according to an embodiment of the present invention. A circle 170, whichhas a diameter given by equation (15), is the circle cutting the planecontaining mask 80A″. A portion 174 of this circle is occluded bycircular mask 80A″, and a portion 178 is transmitted. The expression forthe two-dimensional fraction of occlusion F_(2D) given by equation (16)corresponds to analysis along a line FV′EW′.

There is a corresponding equation for a three-dimensional fraction ofocclusion F_(3D), given by the following expression:

$\begin{matrix}{F_{3D} = \frac{A}{A_{L}}} & (17)\end{matrix}$

-   -   where A is the area of portion 174, and    -   A_(L) is the area of circle 170.

F_(3D) may also be written as:

$\begin{matrix}\begin{matrix}{F_{3D} = {\frac{A}{A_{L}} = \frac{{\alpha_{1}\frac{D_{L}^{2}}{4}} + {\alpha_{2}\frac{d^{2}}{4}} - {M\frac{D_{L}}{2}{\sin\left( \alpha_{1} \right)}}}{\pi\frac{D_{L}^{2}}{4}}}} \\{= {\frac{1}{\pi{p^{2}\left( {1 - \frac{l}{L}} \right)}^{2}}\left( {a{\cos\left( \frac{{p^{2}\left( {1 - \frac{l}{L}} \right)}^{2} + {4\left( {R\frac{l}{L}} \right)^{2}} - d^{2}}{4R\frac{l}{L}{p\left( {1 - \frac{l}{L}} \right)}} \right)}} \right.}} \\{{p^{2}\left( {1 - \frac{l}{L}} \right)^{2}} + {a{\cos\left( \frac{d^{2} + {4\left( {R\frac{l}{L}} \right)^{2}} - {p^{2}\left( {1 - \frac{l}{L}} \right)}^{2}}{4R\frac{l}{L}d} \right)}d^{2}} -} \\\left. {}{2R\frac{l}{L}{p\left( {1 - \frac{l}{L}} \right)}{\sin\left( {a{\cos\left( \frac{{p^{2}\left( {1 - \frac{l}{L}} \right)}^{2} + {4\left( {R\frac{l}{L}} \right)^{2}} - d^{2}}{4R\frac{l}{L}{p\left( {1 - \frac{l}{L}} \right)}} \right)}} \right)}} \right)\end{matrix} & (18)\end{matrix}$

From equation (18), F_(3D) is a function of pupil diameter p, and theequation provides numerical values of F_(3D) for selected values of d,R, p, 1, and L.

FIG. 7 illustrates graphs of occlusion vs. distance, according to anembodiment of the present invention. The graphs have been drawn assumingthe following values:

-   -   L=50 cm    -   l=2 cm    -   P=0.3 cm    -   D=15 cm

From equation (7) the diameter of the occlusion mask to fully occlude anROI with diameter D of 15 cm is d=0.888 cm. The graphs of FIG. 7 havebeen drawn with d set at 0.894 cm.

From equation (15) the diameter of circle 170 is 0.288 cm, so that thevalue of the area A_(L) of the circle is 0.065144 cm².

A solid line graph 200 illustrates the full and partial occlusion vs.distance (from the center of the ROI) for the three-dimensional casecomprising equation (18). The measurements of occlusion have beennormalized, so that for an LCD screen a full occlusion of 95% isnormalized to 1, and a full transparency (of 60% occlusion) isnormalized to 0. A broken line graph 204 illustrates the full andpartial occlusion vs. distance for the two-dimensional case comprisingequation (16). As is apparent from both graphs, there is full occlusion,for a mask of diameter d=0.894 cm, for a region 208 up to approximately8 cm from the center of the ROI, and partial occlusion in a region 212from approximately 8 cm to approximately 15 cm. The fraction of partialocclusion decreases monotonically in region 212.

FIG. 8 is a flowchart of steps performed in operation of system 20,according to an embodiment of the present invention. The steps areassumed to be performed by processor 26 and, as necessary, professional22 for use in a procedure on patient 30 performed by the professionalusing device 38 (FIG. 1 ). In an initial calibration step 300, frame 54is adjusted to center combiners 52 with respect to the eyes of theprofessional. The sizes of the combiners and the distances of thecombiners from the eyes of the professional are also measured andrecorded by the processor (the use of the measurements is describedfurther below). In step 300 assembly 24 is calibrated, i.e. opticalelements of the assembly are registered with each other. Thus, devices68 are oriented on frame 54 to capture generally similar images from theregion in front of combiners 52. If sensor 72 is present it is alsoaligned to capture a corresponding non-visible image from the region.

Processor 26 also orients the images from micro-projectors 64, byregistering the images projected by the micro-projectors onto combiners52 with the scene viewed by professional 22 through the combiners. Theregistration may be accomplished by the professional viewing a scenethrough combiners 52, together with an image of the same scene as it iscaptured by devices 60 and projected by the micro-projectors onto thecombiners. The professional then adjusts the orientation of themicro-projectors and/or the capturing devices so that the projectedimage and the viewed scene coincide.

Typically the registration and adjustment of the micro-projectors andthe capturing devices is performed for different regions of combiners52, such as the left and right peripheral regions, the upper and lowerperipheral regions, and a central region. In addition, the registrationand adjustment may be performed for different scenes according to thedistance of the scene from the combiner, such as a scene of relativelynear elements, typically up to 1 m from the combiner, and a scene ofrelatively far elements, typically greater than 1 m from the combiner.The registrations and adjustments of the micro-projectors and thecapturing devices are typically different for the different regions ofthe combiners, as well as for scenes at different distances from thecombiners. Processor 26 stores the different registration data acquiredduring the calibration step for use when the professional is usingassembly 24.

During the calibration step the sizes of the pupils of the eyes ofprofessional 22 are measured. In one embodiment professional 22 gazes ata circular object of a known diameter and at a known distance from theprofessional, and processor 26 presents an occlusion mask on screens 60to the professional. The professional then adjusts a diameter of theocclusion mask until complete occlusion of the object is achieved. As isapparent from equation (7), the diameter of the completely occludingmask provides a value for the pupil diameter, since d₁, 1, L and D(terms in equation (7)) are all known.

Alternatively or additionally, the professional may look into a mirrorwhile image capturing devices 60 acquire images of the reflected scene,in this case the professional wearing assembly 24. Processor 26 analyzesthe acquired images, by processes that are well known in the art, toidentify the pupils of the professional as well as the outlines ofcombiners 52. The processor then compares the diameters of the pupilswith the known dimensions of the combiners, so as to determine valuesfor the diameters.

The measurements of the pupil diameters are taken for different ambientlight brightnesses, and the ambient brightness values may be determinedfrom the signal levels of the images acquired by devices 68. Processor26 stores the values of the pupil diameters, and the correspondingbrightness levels.

As stated above, processor 26 is configured to track device 38, usingthe one or more identifying elements 39 (FIG. 1 ). In calibration step300 the processor initiates tracking of device 38, and professional 22confirms that the tracking is acceptable.

In an ROI defining step 302, ROI acquisition marker 35 (FIG. 1 ) ispositioned on patient 30, so as to define a region of interest of thepatient selected by the professional, herein assumed to be ROI 34. Asexplained above, marker elements 36 of marker 35 define the position ofROI 34, and the size of the ROI may be defined by the professional.Typically there are at least three marker elements 36, although more maybe used, and characteristics of the elements, such as their color and/orshape, are selected so that they may be easily distinguished frompatient 30. If assembly 24 comprises sensor 72 with an infra-redprojector, marker elements 36 may be configured as retro-reflectorswhich selectively reflect only infra-red radiation.

In an imaging step 304, image capturing devices 68 acquire images of thescene in front of assembly 24. Sensor 72, if present, also captures acorresponding image of the scene. Processor 26 analyzes the images toidentify marker elements 36, and from the identified elements determinesthe orientation of ROI 34 with respect to assembly 24, and also thedistance of the ROI from the assembly. Even if sensor 72 is not present,it will be understood that having two devices 68 acquiring respectiveimages of the scene simplifies the analysis needed to be performed bythe processor to identify elements 36. In addition, having two capturingdevices 68 reduces the number of elements 36 required to accuratelydetermine the orientation and distance of the ROI with respect toassembly 24, compared to the number required if only one capturingdevice 68 is used. With two capturing devices 68 the inventors havefound it is sufficient to have one marker with three marker elements toaccurately locate the ROI with respect to assembly 24. If sensor 72 ispresent, its image alone may be sufficient to identify elements 36,although typically processor 26 uses the images from devices 68 toimprove the accuracy of the orientation and distance measures of the ROIdetermined by the sensor.

Processor 26 also analyzes the images acquired by devices 68 in order todetermine a measure of the brightness of the scene in front of assembly24.

In a frame orientation step 305, the processor rotates combiners 52A and52B with respect to their respective frames so that the combiners areorthogonal to the gaze directions of the professional towards the ROI.The processor uses equations (A), (B) and/or (C) to determine the anglesof rotation of the combiners.

In a masking step 306, the processor generates circular occlusion masks80 in screens 60. The processor, using the orientation of the ROImeasured in step 304 and the central adjustment of combiners 52 in step300, determines positions for the masks that will occlude ROI 34. Fromthe brightness measured in step 304, and from the correspondence betweenpupil size and brightness stored in initial step 300, the processorestimates a value of the pupil diameter of the professional.

In one embodiment the processor sets the diameter of masks 80 accordingto equation (7), i.e., inter alia, according to the professional's pupilsize, so that the masks fully occlude ROI 34. In this case partiallyoccluded ring 130 surrounds ROI 34, the fraction of partial occlusionwithin the ring being given by equations (12) and (18).

In some embodiments the processor determines sections of the scenecorresponding to partially occluded ring 130, and as acquired by devices68. The processor then configures micro-projectors 64 to overlay videoof the acquired sections onto the partially occluded ring, so as tocompensate for the partial occlusion. Typically, processor configuresthe intensity of the projected video to be the inverse of the fractionof the occlusion.

In an alternative embodiment, rather than setting the diameter of themasks to be according to equation (7), the processor sets the diameterto be reduced from the value determined by the equation. The reductionis typically determined by professional 22. In one embodiment thediameter is set to be 90% of the value determined by equation (7).

In a further alternative embodiment, the processor, using instructionsfrom professional 22, sets the diameter of the masks to be larger thanthe diameter of equation (7). In one embodiment the diameter is set tobe 110% of the value determined by equation (7).

In a mask projection step 308 processor 26 uses micro-projectors 64 toproject augmented video onto occlusion masks 80. In the case of theaugmented video including two or more types of images being projectedonto the masks, processor 26 registers the images with each other.However, the images are not necessarily registered, and are typicallymisaligned, with the scene surrounding and outside the masks. Thus, asexemplified by FIG. 2E and the description of the figure, a video image114 of the upper portion of the distal end of device 38, together with astored image 112 corresponding to the lower portion of the distal end,are registered together and are projected onto masks 80. As is alsoillustrated in FIG. 2E, the images on masks 80 are typically misalignedwith the visible scene outside the masks.

In a further projection step 310, processor 26 uses micro-projectors 64to project augmented video onto the partially occluded ring surroundingthe masks, and/or the non-occluded section of combiners 52. As in step308, multiple image types are registered together, but are typicallymisaligned with the visible scene of the non-occluded section.

Typical images that are projected in steps 308 and 310 include, but arenot limited to, those described above with respect to FIGS. 2C, 2D, and2E, and the choice and positioning of the images is typically under theoverall control of professional 22. In a mentoring situation, at leastsome of the images are typically under control of a mentor ofprofessional 22.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and subcombinationsof the various features described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art.

We claim:
 1. A system for conducting an augmented reality assistedmedical procedure on a patient, comprising: a surgical device for use bya medical professional in the procedure, the device comprising one ormore device identifying elements; a region of interest (ROI) markerpositioned in proximity to an ROI; a system processor configured toaccess a database wherein are stored images and other visual elementsrelated to the procedure; and an augmented reality assembly comprising ahead-up display configured to be mounted on the medical professional,the assembly comprising: a micro-projector configured to be operated bya device processor so as to present an image of the ROI; and a capturingdevice configured to be operated by the device processor so as toacquire the image of the ROI; wherein: the capturing device isconfigured to capture an image of the one or more device identifyingelements and of the ROI marker so as to enable the processor to trackthe surgical device and the ROI, the system processor is configured toregister a position of the ROI with a given stored image from thedatabase, and the micro-projector is configured to project an image ofthe tracked surgical device and the tracked ROI with the registeredgiven stored image.
 2. The system according to claim 1, wherein thecapturing device comprises an infra-red camera.
 3. The system accordingto claim 1, wherein the one or more device identifying elements and theROI marker comprise infra-red reflective elements.
 4. The systemaccording to claim 1, and comprising a sensor configured to capture animage of the one or more device identifying elements and of the ROImarker so as to enable the system processor to track the device and theROI.
 5. The system according to claim 4, wherein the sensor comprises aninfra-red camera.
 6. The system according to claim 1, further comprisingan array of pixels, an opacity of each of the pixels being adjustable bythe system processor.
 7. The system according to claim 6, wherein thesystem processor is configured to occlude at least a portion of the ROIby adjusting the opacity of selected pixels of the array, and to presentthe image of the ROI so that the image appears mis-aligned with the ROIby up to a predefined misalignment value, the system processor furtherbeing configured to overlay an augmented reality portion of a sceneviewed by the medical professional on the occluded portion.
 8. Thesystem according to claim 1, wherein the system processor is in theaugmented reality assembly which is mounted on the medical professional.9. A method for conducting an augmented reality assisted medicalprocedure on a patient, comprising: positioning a region of interest(ROI) marker in proximity to an ROI. mounting an augmented realityassembly comprising a head-up display on a medical professional, theassembly comprising: a micro-projector configured to be operated by adevice processor so as to present an image of the ROI; and a capturingdevice configured to be operated by the device processor so as toacquire the image of the ROI; using the ROI marker to register aposition of the ROI with a given stored image from a database; providingthe medical professional with a surgical device for use in theprocedure, the device comprising one or more device identifyingelements; capturing with the capturing device an image of the one ormore device identifying elements and of the ROI marker so as to enable asystem processor to track the surgical device and the ROI; andprojecting with the micro-projector an image of the tracked surgicaldevice and the tracked ROI with the registered given stored image. 10.The method according to claim 9, further comprising providing theaugmented reality assembly with an array of pixels, an opacity of eachpixel being adjustable by the system processor.
 11. The method accordingto claim 10, and comprising occluding at least a portion of the ROI byadjusting the opacity of selected pixels of the array, and presentingthe image of the ROI so that the image appears mis-aligned with the ROIby up to a predefined misalignment value, and overlaying an augmentedreality portion of a scene viewed by the medical professional on theoccluded portion.
 12. The method according to claim 9, wherein thecapturing device comprises an infra-red camera, and wherein the one ormore device identifying elements and the ROI marker comprise infra-redreflective elements.
 13. The method according to claim 9, and comprisingproviding a sensor comprising an infra-red camera, and wherein the oneor more device identifying elements and the ROI marker compriseinfra-red reflective elements.